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ABSTRACT 

This sample evaluation report is part of a series of 
resources developed for and by workplace education practitioners in 
business, education, and labor partnerships funded through the 
Massachusetts Department of Education's Workplace Literacy Program. 
Data included in the report are based on Project Health, which 
integrates the experiences from several workplace education programs « 
Much of the data and background information are adapted from 
publications produced by the South Cove Manor Nursing Home Workplace 
Education Program in Boston's Chinatown. Section I describes the 
program background. Section II outlines the evaluation process, 
provides the evaluation timeline, describes the process of 
identifying goals and indicators, and presents the work plan that 
details the selected goal and corresponding indicators/outcomes, 
information needed, instrument to get information, and partner. 
Section III describes data collection, instrument design, and field 
testing. Section IV presents evaluation results that reflect only a 
small segment of the opportunities that the Statistical Package for 
the Social Sciences offers in regard to data analysis. Section IV 
lists areas for improvement and reflections on the process. 
Attachments include instruments and flow sheet exercises. (YLB) 
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September, 1993 

I am pleased to present this publication developed through our Massachusetts Workplace 
Education Initiative. This publication is part of a series of resources developed for and by 
workplace education practitioners in business, education, and labor partnerships funded 
through our Department's Workplace Literacy Program. 

These resources are the result of our commitment to strengthening the capacity, knowledge 
base, and quality of the field and to provide much-needed and long-awaited information on 
highly-innovative and replicable practices. These resources also complement the curriculum 
framework of staff training and development initiatives that were successfully developed and 
piloted in conjunction with the field during the past fiscal year and represent an oustanding 
example of the Department's theme: " Working Together for Better Results." 

Each of these publications was written by trainers and workshop presenters who have 
participated in the training of new workplace education staff. All publications provide 
invaluable information on important aspects of workplace education programming. All 
documents begin with an overview of the field or current-state-of-the-art section as it relates 
to the topic at hand. Then, they move into the practitioner's experience. Next, the training 
plan of presenters is discussed. Each publication ends with a list of resources. 

We are confident that with this series of publications we have begun an exciting but 
challenging journey that will further support workplaces in their progression towards 
becoming high-performance work organizations. 



Sincerely, 




Robert V. Antonucci 
Commisioner of Education 
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EVALUATION REPORT 




INTRODUCTION 



This evaluation report is a sample report. It was developed for training purposes. It is our intent that this publication 
will meet the need for guidance that several projects have expressed when they considered documenting the 
outcomes of their evaluation work. When using this resource as a possible model, programs should be aware that 
their reports should not necessarily include an equal amount of background information. They should also be aware 
that the report cnly reflects a small segment of the opportunities that SPSS. PC (Statistical Package for the Social 
Sciences) offers in regards to data analysis. 

The data included in the report are based on Project Health. Project Health integrates the experiences from several 
workplace education programs funded by the Massachusetts Department of Education through the National 
Workplace Literacy Program. However, much of the data and background information are taken and adapted from 
publications produced by the South Cove Manor Nursing Home Workplace Education Program in Boston's 
Chinatown. 
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L PROGRAM BACKGROUND 



WHY SUNSET LTC 
NEEDED A WORKPLACE ESL 
PROGRAM ? 



□ In the Winter of 1990 the Board of 
Directors of the Sunset Long Term Care facility 
in Boston's Chinatown appointed a Patient Care 
Services Committee to respond to the need for 
ESOL instruction (English to Speakers of Other 
Languages) for its staff. The committee was 
assigned to carry out an informal organizational 
needs analysis and to clarify the relationship 
between the quality of patient care and the 
ability of staff to communicate in English. 

□ Upon completion of its task, the Patient 
Care Services Committee concluded that 
Sunset Long Term Care needed to strengthen 
its ability to deliver high-quality patient care. 
The members of the committee identified a 
direct negative relationship between the quality 
of patient care and the ability of monolingual 
Chinese and Haitian caregivers to communicate 
with English speaking residents, 

with visitors, with supervisory, licensed, and 
administrative staff, with management, and 
with each other. Interpreters and translations 
were needed to facilitate communication and 
care was too often provided without any verbal 
explanation of procedures. 

In addition, the members of the Patient Care 
Services Committee attributed the 
compromised quality of care to the problem of 
illiteracy in English. Nursing assistants and 
entry-level workers in the dietary, activity, and 
maintenance departments did not have the 
reading, writing, computational, and problem- 
solving skills to perform their tasks 
satisfactorily. Unless there was on-going 
bilingual assistance, workers faced considerable 
problems in reading and completing patient 
care documents. 

The committee also acknowledged that Sunset 
Long Term Care experienced difficulty in 
recruiting and retaining licensed staff. While 
many of the facility's non-professional staff had 
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higher-level nursing experience in their 
countries of origin, Sunset Long Term Care was 
unable to employ these underemployed health 
care professionals because their diplomas and 
degrees were not recognized by the U.S. 
Government. While aware of the relatively long 
learning time required, the nursing home hoped 
that some of these highly-skilled workers could 
develop the necessary language skills to enroll 
in and eventually successfully complete nursing 
programs or take licensing examinations, which 
in the long term would make them eligible'to fill 
licensed staff positions. 

Another factor that impacted on the quality of 
patient care was the high turnover amongst 
entry-level caregivers (e.g. 56% in 1989). As 
elsewhere in the Commonwealth throughout 
the eighties, Sunset Long Term Care was 
facing a high turnover rate. As a result, the 
continuity and quality of care were frequently 
at risk. The nursing home, consequently, 
needed to recruit more expensive temporary 
agency staff and its recruitment and training 
costs went up considerably. 

The job awareness and readiness of some 
employees further compounded the problem. A 
sizeable group of nursing assistants - mostly 
recent arrivals — demonstrated a limited 
understanding of American culture and rapidly 
changing approaches to health care delivery in 
the United States. Their perceptions of quality 
care were not always compatible with those of 
residents and the nursing home management. 
Although with good intentions, staff often 
responded to the needs and expectations of 
residents in ways that were not always 
culturally-appropriate. 

Finally, the implementation of the new 
certification requirements for nursing assistants 
added yet another dimension to the problem. 
While most of the staff were allowed to take 
the skills and written sections of the National 
Nursing Assistant Examination in their first 
language based on the make-up of Sunset's 
resident population, the non-Chinese staff 
faced an additional challenge that needed to be 
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addressed. Most of the Haitian staff, for 
example, experienced some difficulty with the 
vocabulary used in the written section of the 
exam and the testtaking strategies required to 
successfully complete the test. 

□ In the Spring of 1991 the administrator 
conducted an employee survey to ascertain the 
level of need and degree of employee interest. 
More than fifty or half of the nursing home's 
staff responded and expressed a strong need 
and commitment to participate in an on-site 
Workplace ESOL program. 

□ In response to the Administrator's report, 
the Patient Care Service Committee 
recommended to the Board that Sunset Long 
Term Care employ multiple strategies to 
address the issue of compromised care and 
clarified that more would be needed than a 
workplace education program. In addition, the 
committee suggested a long-term education 
and training plan be put in place consisting of 
a multi-pronged strategy to address the 
educational needs of different groups of 
individuals on staff. 




WHO WERE THE PARTNERS IN THE 
SUNSET LTC WORKPLACE 
EDUCATION PROGRAM ? 



□ To respond to the broad range of 
educational needs, Sunset Long Term Care 
contacted the Chinese Community Action 
Council to collaborate on the development of a 
Workplace ESOL/Literacv program. A 
partnership was formed and Project Health was 
established in the Summer of 1992 with 
funding ($80,000) from the U.S. DOE National 
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Workplace Literacy Program through the 
Massachusetts Department of Education. 
Funding was received for 1 8 months. 

□ A Planning and Evaluation Team was 
established consisting of the Administrator, the 
Director of Nursing, the Staff Development 
Coordinator, Morning, Afternoon, and Evening 
Shift Supervisors, the Project Coordinator and 
Instructors, and two employees from each 
shift, one recently-hired employee and one with 
some tenure. Employees were nominated by 
their peers. All team members received a three- 
hour orientation. This orientation involved an 
orientation to the program and to the role of 
the team. More specifically, employees were 
trained how to solicit and report input from 
their peers, how to participate actively in 
meetings, how to ask for clarification and make 
suggestions, amongst other skills. To ensure 
communication at team meetings, interpreting 
services were made available. To balance 
participation, the team decided that the multi- 
lingual Project Coordinator should facilitate the 
initial meetings. After 3 meetings, team 
members took turns in chairing the meetings 
and notetaking. To assure program 
responsiveness to the needs of all those 
involved, consensus was identified as the 
decision making mode. 

Not including data collection, the Planning and 
Evaluation Team devoted 36 hours of meeting 
time to planning, implementing, evaluating, and 
monitoring the program. During the first 6 
months meetings were held every three weeks. 
Later into the project meetings were scheduled 
less frequently. The Project Coordinator and 
Instructors spent a total of 48 hours on the 
design of data-gathering instruments, the 
collection and analysis of data, and the 
preparing of reports. 





SUNSET LTC 

The Sunset Long Term Care facility (LTC) is a 
100-bed nursing home in the South End of 
Boston on the edge of Chinatown, ft is one of 
three long-term care facilities dedicated to 
serving the Chinese elderly, and the only 
located on the East Coast. Since November 
1985, Sunset LTC has provided care to one 
hundred residents in a homelike environment. 
To sponsor its operations, the nursing home 
depends largely on Medicaid funds. Most 
recently, the home qualified as a Medicare 
recipient. In addition, Sunset LTC receives 
generous support from the Chinese community, 
from corporations, from foundations, and from 
resident families. 

More than 50% of the residents are Chinese. 
Approximately 40% speak English only, with 
the rest speaking one or more of several 
Chinese dialects. While some Asian residents 
speak English, most do not. 

About one hundred employees are on staff. 
More than 90% are women and the vast 
majority are Asian immigrants. Employment 
opportunities exist in direct and indirect 
caregiving roles. The largest department is the 
nursing department. All staff are trained in 
geriatric nursing. The maintenance department 
keeps the plant clean and safe and provides 
laundry services. The dietary department 
provides food services to meet the dietary 
needs of residents. The Activities Director and 
her aidefs) provide recreational opportunities 
and occupational therapy for residents. A 
social worker acts as the liaison between the 
resident, the resident's family, and the facility. 

In addition to in-house staff, several other 
health care providers deliver care at Sunset. 
These include physicians, physical and 
occupational therapists, dieticians, dentists, 
pharmacists, and others. 



WHAT WERE THE GOALS AND 
COMPONENTS OF THE PROGRAM ? 

GOALS 

The Planning and Evaluation Team felt that a 
gradual approach to developing a 
comprehensive education program for all 
employees was feasible. Its first step was 
Project Health named by employeas through a 
contest. Project Health did not aim at trying to 
meet the needs of all employees during the first 
18 months of the program. It was designed to 
serve the most needy, that is, the 29 
monolingual Chinese-speaking nursing 
assistants who on their jobs needed to interact 
frequently with English speaking residents. This 
commitment to serve the most needed aligned 
well with the nursing home's 's overall mission 
to provide, improve, and assure restorative care 
of high-quality and translated in the following 
overall proaranvgoal : 




1. 




improve the quality of patient care by 
enhancing the basic skills of all 
monolingual Chinese-speaking nursing 
assistants so that they acquire those 
skills that are critical to the delivery of 
patient care. 



The nursing home representatives on the team 
hoped that the program would also 



promote job retention and reduce hiring 
and training costs associated with 
temporary staff; 

support interested graduates in 
applying for opportunities for further 
training and education; 
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COMPONENTS OF THE PROGRAM 



After the Planning and Evaluation Team had 
identified the goals and objectives for the 
program, the team listed which components 
needed to be put into place for Project Health 
to achieve its goals. The team devoted several 
meetings to this task and developed or refined 
the following key components: 

20 

1. A mission statement and description of 
program goals and philosophy to be used in 
outreach, recruitment, and program 
orientations; 

2. Clear role descriptions for the Planning and 
Evaluation Team; 

3. Clear job descriptions for staff; 

4. Clear diagram of channels of 
communication and decisionmaking 
processes; 

5. Criteria to select qualified staff who are 
sensitive to needs of the Asian immigrant 
population; 

6. Program Orientations for residents, potential 
participants, supervisors, department heads, 
administrators, and managers; 

7. Outreach and Recruitment Plan to identify 
participants using linguistically- and 
culturally-appropriate strategies and 
materials; 

8. Referral plan to place applicants a 

and graduates inappropriate programs to be 
developed at Sunset LTC or elsewhere if 
their needs move beyond the scope of the 
program; 

9 . Memoranda of Agreement that demonstrate 
linkages with individuals at Sunset LTC or 
with agencies in the community to ensure 
access to support services such as counseling 
and childcare, as well as 
opportunities for further training 
and learning; 



10. A basic skills analysis to identify which 
basic skills are critical to the delivery of 
patient care; 

1 1 . Appropriate and convenient learning 
arrangements, Workplace ESL classes and 
educational/career counseling sessionsin 
this instance, to facilitate the teaching and 
learning of English and promote 
adavancement; 

12. Intake procedures that facilitate language 
assessment, appropriate placement and/or 
referral; 

13. Individual Education Plans with both short- 
term goals (for the program) and long-term 
goals; 

14. Process for curriculum development and 
documentation that integrates learner, 
teacher, and company input on an on-going 
basis; 

1 5. Assessment tools to collect baseline 
information, as well as data on participant 
progress; 

1 6. Evaluation procedures to collect information 
in the areas of participant assessment, 
transfer of learning into improved job 
performance, and possible changes in 
organizational performance (e.g. work 
retention rates and quality of care); 

17. Accurate recordkeeping and reporting 
mechanisms for enrollment, attendance, 
class and work retention, access and 
utilization rates, and demographic 
information; 

1 8. Appropriate facilities (i.e. space) and 
resources; 

19. Clear policies regarding release time, 
attendance, use of Sunset LTC facilities and 
materials; and Sunset LTC personnel 
policies. 

20. Institutionalization plan. 
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II. IDENTIFYING GOALS AND INDICATORS 



The Process 



The Planning and Evaluation Team designed its evaluation as an integral part of the planning 
process. The first step in the evaluation process was to decide which approach would be used. The 
team agreed on the use of a collaborative approach to evaluation that would be consistent with the 
its program development approach and that would integrate the needs of all partners involved, 
speak to their preferences and resources, and develop the capacity of Sunset LTC to carry out its 
own evaluation. 

To facilitate this process, the team also requested the support of the Massachusetts Department of 
Education who provided the services of a Resource Person, two training sessions — one in 
teambuilding and evaluation and one in data analysis — and several statewide sharing sessions. The 
team adapted the process introduced by the Massachusetts Department of Education as follows: 



STAGE 


TIME THAT WAS REQUIRED 


CALENDAR 


REFINE GOALS AND SELECT 
WHICH ONES TO EVALUATE 


3 HOURS 


JULY 1992 


IDENTIFY INDICATORS OF 
SUCCESS/QUALITY AND 
ANTICIPATED OUTCOMES 


3 HOURS 


JULY 1 992 


VERIFY IF ALL NECESSARY 
COMPONENTS ARE IN 
PLACE 


3 HOURS 


AUGUST 1 992 


IDENTIFY EXISTING 
SOURCES OF INFORMATION 
AND DATAGATHERING 
INSTRUMENTS 


3 HOURS 


AUGUST 1992 


DESIGN AND REFINE NEW 
DATAGATHERING 
INSTRUMENTS 


9 HOURS 


AUGUST/SEPTEMBER 1992 


COLLECT DATA 


ON-GOING 


SEPTEMBER 1 992/ 
DECEMBER 1993 


ANALYZE DATA, PREPARE 
AND SH *RE REPORTS 


9 HOURS 


UPON CONCLUSION OF 
CYCLES 


DEVELOP AND IMPLEMENT 
ACTION PLANS TO IMPROVE 
PROGRAM 


3 HOURS 


CYCLE BREAKS 
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Three focus groups were established to refine the goals of the program and to identify indicators of 
success and/or quality. One group consisted of supervisors and the administrator. The second group 
consisted of learners in the program. This group was facilitated by the bilingual ESL Instructor. The third 
group was made up of the Project Coordinator, Instructors, and Sunset LTC's Staff Development 
Coordinator. Focus groups met separately twice for 1 hour and once for an additonal hour as a large group. 
In their meetings, focus groups made sure everyone reached the same level of understanding of the 
program goals identified by the Planning and Evaluation Team and made revisions as necessary. After that, 
each group listed indicators that would show that the Project Health was moving towards achieving its 
goals. These indicators were defined as signs of "success or quality". Subsequently, focus groups 
identified which information was already available or needed. Their last task involved brainstorming ways 
to collect information and drafting a workplan. 




The workplan below integrates the findings and recommendations from all focus groups and reflects the 
consensus reached at the large group meeting. 



SELECTED GOAL 


INDICATORS 


INFO NEEDED 


INSTRUMENT TO 


PARTNER 




OUTCOMES 




GET INFO 




GOAL #1: 










Enhance the basic skills of 29 


AH employees 


Number of eligible 


Review of program 


Administrator 


monolingual Chinese-speaking 


whom Project 


employees (i.e. 29) 


and company 




nursing assistants who need to 


Health wants to 




records 




interact in English on the job 


serve can access 


Number of enrolled 




Project Director 




the program 


employees who are 










eligible 








The highest 










possible number of 


Class attendance and 








employees in need 


class retention data 








of the program 










enroll voluntarily 










and stay with the 










program 










throughout the 










grant period 
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OtLtU 1 CU uUAU 


INDICATORS 


INFO NcEPEP 


IUCTDI t UCtIT TA 

INSTRUMENT TO 


PARTNER 




OUTCOMES 




APT ItltA 

GET INFO 






Participants better 


inventory of Critical 


Learner Self- 


I nRtn i/*tnr 

II loll Uw IUI 




master the skills 


Skills 


Assessment 






that are critical to 






Participants 




thft HpJix/ftrv nf 
11 iu VJvuvui y vi 










high-quality patient 


Participant data that 


Classroom-Based 


Supervisor 




care 


show increased 


Simulations 








proficiency level based 




Project Director 






on inventory 












Resident interview 










Supervisor 










Questionnaire 




GOAL #2: 










PramnfA AmnlnvA^ r*t*ntimn 


RptAnfinn ratp nf 


RAtpntinn ratAc nf 
ncicuuuii idiuo ui 


\*v i ii iuci i iioi review 


HU I Til nioli alUi 




participants vs non* 


nartipinnntR nnri nnn- 


nf pmnln\/pp 






nnrtipinantc 


na rtipinnntc in Mntrv/- 
\jo i ui*i}j<ii 1 io hi oiiii y 








improves 


level nursing assistant 




Project Director 






positions 










Info on other initiatives 










taken by Sunset LTC 










and their impact on 










retention (e.g. improved 










benefit package) 






GOAL #3: 










Reduce costs related to hiring 


Comparison of 


Budgets and spending 


Confidential review 


Administrator 


and training of tamporary staff 


projected and 


plans Sunset LTC 


of employee 






actual costs for 




records 






hiring and training 










of temporary staff 


Financial quarterly and 








pre, during, and 


annual reports 




Project Director 




post program 








GOAL #4: 










Support intarastad graduates in 


Graduates request 


Number of graduates 


Learner Self- 


Learners 


applying for opportunitias for 


information about 


who request 


Reported Data 




furthar aducation and training 


further education 


information, apply, and 






at Sunaat LTC and elsewhere 


and training at 


enroll in aducation and 






in tha community 


Sunset LTC and in 


training programs at 


Teacher Follow-Up 






community 


Sunset LTC and in 


with learners and 


Instructors 






community 


agencies 






Graduates apply 










Graduates enroll 










III. COLLECTING DATA 



□ In the selection of methods and instruments 
to gather data, the team made a conscious effort 
to build on existing systems such as Sunset LTC 
records and program intake forms. However, 
these were not sufficient in that they could not 
generate data for each goal; nor could they 
document any unanticipated outcomes. 

□ In developing new instruments the team 
based its work on the following assumptions : 
Instruments should be reliable and valid, and 
therefore, should be fiefdtested carefully before 
implementation. Instruments should also generate 
site-specific data and strike a balance between 
qualitative and quantitative information. The team 
also felt that the instruments should be designed 
collaboratively and should involve learners as 
actively as possible. The team also wanted the 
instruments to be cost-effective and easy and 
quick to administer. Finally, given the grant 
guidelines, the team clarfied that instruments 
should mainly aim at gathering work-related 
information. Another important belief of the team 
was that 



all records should be 
kept confidential 



and that the diverse, preferred assessment 
activities of learners should be taken into 
account. To do so, instructors asked learners in 
all classes how they wanted to find out how they 
were making progress and which kind of 
information or feedback would be most useful to 
them. 

Building on these principles, the following 
instruments were designed and fieldtested: 

- a resident interview 

- a learner self-assessment questionnaire, 

- a questionnaire for supervisors, nurses, 
and managers 

- a series of classroom-based simulations, 

- a classroom-based flow sheet exercise, 

- a program log. 
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Copies of the learner self-assessment, the 
resident interview, and the questionnaires are 

included in the Attachments of this report. 

The simulations were based on the Inventory of 
Critical Skills (See Attachments) in the Delivery of 
Patient Care. This way there was a one-to-one 
correspondence between the skills included in the 
learner self-assessment and the simulations. 
During the simulations, learners were asked to 
perform an actual job-related task and were 
observed and assessed by their peers and 
teachers. For each of the simulations a number 
value was attached to the response of learners 
ranging from 1 to 3. For each of the learners, 
records were kept that showed how learners 
were making progress towards achieving each of 
the key skills so that these data could inform 
future instruction and curriculum. In the data 
analysis stage, however, only the percentage of 
skills achieved was used. 

The flow sheet exercise provided learners with a 
written or taped description (both available in 
English and the learners' first languages) of a 
common set of tasks to be performed. Learners 
were asked to enter the information on the flow 
sheet. The percentage of items that were entered 
correctly was used in the data analysis. 

Once the instruments were fieldtested, the team 
revisited its workplan and assigned team 
members to various data collection tasks. A total 
of 48 hours was spent on drafting, fieldtesting 
and using the instruments and the analysis of 
data. All data collection took place either on class 
or company time. 
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IV. RESULTS 



PROJECT HEALTH IS 
100% ACCESSIBLE 

□ Classes were scheduled so that all 
29 eligible employees could attend if 
they so desired. 

UTILIZATION RATE 
IS OVER 85% 

□ 25 of the 29 eligible nursing 
assistants enrolled in the program 
during the first cycle. Except for two, 
all were Asian women whose first 
language was Cantonese and who 
completed at least high school in 
China, their country of origin. 

□ A slot level of 25 was maintained 
throughout the grant period. In 
December 1993, more than 85% of 
all eligible employees enrolled 
voluntarily in the Workplace ESOL 
component of the program. According 
to participants, the program was 
successful because class time and 
location were convenient, because the 
curriculum was responsive, and 
because teaching was of high quality. 




A TTENDANCE WAS ALARMING 
IN FALL CYCLE 

□ The average attendance rate for Cycle 1 and 
Cycle 2 were comparable and in the 75-80% 
range. Rates for Cycle 3, however, were fairly low 
According to the team's findings this is mostly due 
to staff vacations. 



Att«nd«nc« R«U P«r Month 




LEARNERS MADE PROGRESS 
AND FEEL SELF CONFIDENT 

JOB PERFORMANCE IMPROVED 

QUALITY OF CARE LESS 
COMPROMISED 

RESIDENTS MORE SA T/SFIED 

GRADUATES ENROLLED IN 
NURSING PROGRAMS 
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IV. RESULTS 



LEARNER PROGRESS IS 
EVIDENT 

□ Based on comparable data from 19 
participants, it is clear that the 
average percentage of skills attained 
has and will continue to go up. Based 
on the available data, the team 
inferred the trend that on average and 
with the current program design 
employees will need a total of 
approximately 5 cycles of instruction 
to achieve all key skills identified 
including the 4 already offered. 

□ The classroom-based simulations 
have proven to be reliable instruments 
in assessing the 30 oral 
communication skills that are critical 
to the delivery of patient care. 



Av«r«g« of K«y Skill* 
Attained 




Poil Program 



□ Flow Sheet Exercise scores 
indicate a similar trend for key reading 
and writing skills in the 
documentation of patient care. 



Histogram Simulations 
Assessment Prior to Cycle 1 
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IV. RESULTS 




ABILITY TO 
DOCUMENT 
WORK IMPROVED 
CONSIDERABLY 



%%»t*m Flew %h—\ 
f xarclaa 




Dependence on 
Interpreting 
Down in 2/3 of All 
Interactions 

□ 2/3 of English-speaking 
staff do no longer require an 
interpreter to communicate 
with participants in 67% of 
all their interactions. 




QUALITY OF 
CARE IS 
IMPROVING BUT 

LINK WITH 
PROGRAM IS NOT 
CLEAR 

□ The conclusion that 
the quality of care is 
improving is based on 
the belief that 
improved reporting 
skills enable nursing 
staff to make the 
necessary adjustments 
to patient care plans, 
which in turn could -- 
depending on the 
resident's condition -- 
but will not necessarily 
lead to improvement in 
the patient's condition. 

□ While all supervisors 
and nurses agreed that 
improved oral 
communication skills 
have enhanced the 
quality and frequency 
of interactions 
between program 
participants and 
residents, only 65% of 
all reports to the 
Director of Nursing and 
50% of all reports to 



monolingual English-speaking 
nurses and supervisors have 
enabled Sunset LTC to adjust 
patient care plans (as opposed 
to 91 .6% of all reports to 
bilingual nurses and 
supervisors). 

□ According to an analysis by 
the Director of Nursing, these 
adjustements have led to 
improvements in the resident's 
condition in about 25% of all 
cases. 

11% OF 
GRADUATES 
ENROLLED IN PRE 
NURSING OR LPN 
PROGRAMS 
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IV. RESULTS 



TRAINING COSTS 
DROPPED WITH 
MORE THAN 70% 
AND HIRING 
COSTS WITH 
MORE THAN 60% 

Training costs dropped from almost 
$73,000 in the fiscal year prior to 
the program (FY'92) to about 
$17,000 during the program with 
FY'93 costs a little bit lower than 
costs during the first Half of FY'94. 

Hiring costs decreased following 
the same pattern. Annua! costs 
prior to the program exceeded 
$ 3,000, dropped to about $1,000 
in FY'93 with a slight increase in 
the first 6 months of FY'94. 



Training and Hiring Ccsts 



0.000 . 
2.000. 
4.000. 
•.000 
•.000 
0.000 
2.000 
4.000 




m 



FY <B PRE FY«0 




| Hiring CMi Q Training Cotts Q Total 



According to the Planning and 
Evaluation Team, reduced costs are 
a direct result of the improved job 
retention rate. 



JOB RETENTION 
AT RECORD HIGH 



90 + 



Job Retention improved 
considerably and stabilized 
round the 90% mark for 
participants. 




gj Participants q Olnars » Total 

According to the Planning 
and Evaluation Team, the 
positive change is due to a 
number of initiatives 
including the program , an 
improved benefit package 
which was implemented 
simultaneously, and the 
changing economic climate 
which resolved the labor 
shortage. 

The Team concluded, 
however, that the program 
did play an important role 
based on the slightly higher 
job retention rates for 
participants. 



CLASS RETENTION 
INDICATES NEED TO 
REVISIT DESIGN 

Class Retention moved beyond the 
team's expectations. The 80% target 
set by the team for each cycle was 
exceeded in Cycle 1 (95%) and Cyde 
2 (93%). A drop of more than 1 2% in 
Cycle 3 (79%) was recorded. 
According to an informal survey by 
the Project Coordinator this decrease 
is due to staff vacations . The team 
team concluded that the cycle 
schedule needs to be revised. 



Class Retention 
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IV. RESULTS 



UNANTICIPATED OUTCOMES 

Participants also identified the following outcomes that are not goal-related: 

... USE ENGLISH FOR JOB HUNTING *** UNDERSTAND MORE 
LABELS WHEN SHOPPING *** ASK AND GIVE DIRECTIONS ON 
THE STREET TO STRANGERS *** LIVE IN USA WITH MORE 
CONFIDENCE *** UNDERSTAND AMERICAN WAY OF LIFE BETTER 
*** KNOW MORE ABOUT WHAT IS HAPPENING IN THE NATION 

AND THE WORLD *** FEEL READY TO MOVE ON TO MORE 
EDUCATION *** WANT TO TAKE TOEFEL EXAM AND GO BACK 
TO SCHOOL *** FEEL MOTIVATED TO LOOK FOR A BETTER JOB 

*** FEEL BETTER ABOUT HELPING MY CHILDREN WITH 
HOMEWORK *** UNDERSTAND DIFFERENCES BETWEEN NURSING 
IN CHINA AND USA *** READ AND WRITE IN ENGLISH FOR THE 
FIRST TIME SINCE ARRIVAL *** FEEL BETTER ABOUT MYSELF *** 
ORDER FOOD AT RESTAURANT INSTEAD OF HAVING CHILDREN 

DO IT *** GO TO INS WITHOUT HELP OF CHILDREN *** ASK 
OFFICERS AT BOSTON HOUSING AUTHORITY IF FORMS WERE OK 
*** DISCUSS PROBLEM WITH BUILDING MANAGER DIRECTLY ... 

The Director of Nursing also said: 

" ... Our daily Patient Care Rounds have become a lot more productive particularly 
those where nursing assistants are involved who have been in the program for 
some time. There are less misunderstandings and we seem to get each other's 
points faster ..." 

An English-speaking nurse said: 

" ... Since the program started I have learned how much it takes to learn a new 
language. I have really developed a new sense of appreciation. I don't know if I 
could do it..." 

One supervisor noted: 

" Sunset LTC has become a different place. People have opened up. It's just much 
nicer to be here. Before, I remember, I used to sometimes hate to come in because 
I knew what communication barriers I would face. For me it has really made a 
difference and I'd love to go visit China ..." 



IV. REFLECTIONS 



AREAS FOR 
IMPROVEMENT 



Is 



4. 
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□ NEXT TIME 

1. Spend less time as a 
team on drafting 
instruments and 
evaluation overall. 

2. Cancel meetings if not 
all are present. 

3. Follow our own process 
and progress at our own 
pace BUT complete 
collection of baseline 
data before instruction 
resumes . 

4. Find better ways to 
make meetings more 
meaningful to managers 
and learners. 

5. Do teambui lding 
exercises first. 

6. Orient team members on 
how to solicit input from 
their peers. 

7. View evaluation as an 
integral part of our 
team's agenda and not as 
a separate one. 
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ATTACHMENTS 



Change* in functional U*e* of English in the Delivery of Patient Care 



This list vas generated by participants, supervisors, charge and medication 
nurses and stair development coordinators. They vere asked to identify those uses or 
English that contribute directly to the quality of patient cere. In a second phase, 
they vere asked to identify the most essential uses end coded them using the letter e . 



of Orel language ^ 

P.esijentstaregiwr Intersections 

□ knocks end asks for permission to enter | 

□ greets resident using name and/or preferred form of address t 

□ identifies self by neme end job (e.g. I'm Johan. I'm your nurse 

aide today) z. 

□ offers help end services r 

□ states reason for coming /vhat Tall happen L 

□ explains procedures 

0 instructs resident speaking clearly and slovly (e.g. vhile reeding. 

toileting, ambulating, etc.) | 

□ asks about toileting needs £: 

□ encourages resident to eat, exercise, etc. | 

□ indicates lack of understanding r 

□ apologizes if necessary £: 

□ asks for assistance 

□ ends convesation inabruptly 

□ gives varnings in case of danger 

□ directs residents to people /rooms 

□ responds to resident requests, concerns, and complaints 

(e.g. call light) t 

□ maintains eyecontact as much as possible 

□ initiates and maintains a conversation vhile providing care 

to promote comfort end vell-being 

□ asks for various kinds of feedback (e .g . comfort, appetite, clothing 

preferences, order of feeding, bath vater temperature, etc J t 

CaregiversSv^rrtsoryanc" Licensed Starr Interactions 

v 

□ asks for clarification of instructions t 

□ f olio vs instructions of charge nurse, Director of Nursing, etc . t 

□ follovs instructions over P.A.-system including emergency 

□ reports changes in condition orally (e.g. at patient care rounds 

or vhen leaving shift) . .„„ 

□ relays resident messages to appropriate licensed or supervi^o. y ^ 

□ reports*accidents, fells, unsafe conditions, abuse, mistreatment, ^ 

□ reports breakdowns end shortages end states need for materials z 

□ participates actively in Nurse's Aides' Meeting end In-services 

□ trains a nev co-worker 
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a arcs to oe excuses rrom ™* 

□ fcnov: hov to cell in sick or late E 
D asks for training 

Interactions vita Other Departments 

D asks to be rescheduled 

□ asks for time off . 

□ asks about pay. benefits, end rights 

□ asks for meal changes for resident 

□ asks for a recommendation 

□ asks about job openings 

□ asks f or a raise 

ysfcs of Lit eracy and Numeracy 

r 

□ infers vorl: time from schedule 

□ uses time card according to policies 

□ infers instructions for serving from diet cards 

□ enters information about completed tasks and observations onto 

English flov sheet 

□ fills out accident report forms in English ^.,^ t . 
□.fol^ and posted instructions 

(safety, infection control) . AiM „ n0 ... 

□ locates names, rooms, and other information on resident directory 

□ inferf fnstruct^ns from English assignment sheet using reference 

skills to locate information that is needed 
0 infers additional tasks and responsibilities from daily posted £ 
nursing sheet 7 

□ measures and records vital signs end intake and output 

□ documents collection of specimen ^^^tnrm* 

□ fills out time English change sheet and vacation request forms 

□ reports abuse, mistreatment, and neglect in -writing 

□ asks forlSdmaxion about training, licensing, and education 

programs 
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Int ervi evGuid elin e* 
Resident Interview 



Nam e 0/ resi d ent : pl 0 cr: . 

Name of participant: Your name: m Dale:. 



INSTRUCTIONS: Explain the purpose of theinterviewtotheresident(e.g. D^i wants to find out how 
well nursing assistants can use English to talk vith you") Ask if it is fine that you would take notes. 
Mention that the results of the interview \a\1 be kept confidential. Use one ormore of these questions to 
initiate ormaintain a brief conversation. Record the resident's answer by checking the answerthat comes 
closest to what the resident says. Feel free to record responses more elaborately 



1. Does he/she speak English with you during bathing, dressinct, 
eating, toileting, or walking ? 

0 Yes 0 No 0 



2. Does he/she explain what he/she will be doing with you ? 
0 Yes 0 No 0 



3. Does he/she give your messacres to the nurse ? 
C Yes 0 No 0 



4. How well do you understand him/her when she/he is talkina 
with you ? 

0 not at all 0 a little O well 

0 

5- How well does he/she understand you when you are talkincr to 
him/her ? 

0 not at all 0 a little 0 well 

If 4 + 5 are hard, why do you think that is ? 

0 doesn't know Enqlish very well 
0 doesn't understand her job 

0 

6. Overall, has talking with him/her 

0 gotten better ? 0 crotten worse ? 0 stayed about the same ? 



COMMENTS 
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Questionnaire to be completed by English speaking 
supervisory and licensed staff 



Your position: xJav'^e Dale: A (2l* / Name of nursing assistant? 




PART 1 

Reporting changes in residents* condition 

1. Do you share a language with him/her other than English ? 

0 Yes b Vio 
If yes, which one(s) ? 



2. How long have you worked with him/her ? 

CP^- < f / ' Alr 

3. Does he/she report to you 

0 directly in English ? 
O directly in Chinese ? 
0 through an interpreter ? 
0 not at all 

4. How often does he/she report to you ?(e r g. once a week during 
patient care rounds) u k yy drf-ir'-^ ? r f . ^^jP 

5. Have his/her reports enabled you to adjust the patient care 
plan as necessary ? 



Yes 00 Mo 0 




If yes, have these adjustments had an impact on the resident 1 
condition ? / 

^j/bs 0 No 0 



Please, explain your answer: /CfXrT^v^ ^vt)b 



6 . Do you interpret for him/her ? 

0 Yes 0 No 

If yes, how often ? (e.g. once a week) ^tP^C^r^ v^J^ _ 

7. Would you say he/she has become less dependent on you to 
interpret for him/her ? 

r£s 0 No 0 
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8, Do you need an interpreter or written translations to instruct 
him/her ? 

C^Yes 0 No 0 Not usually 
If yes, under what circumstances yui^^x r_ ^Ur v — ^ 



9. Have you become less dependent on interpreters or translations 
co communicate with him/her ? 

es 0 No 

10. Please, record any other comments related to his/her ability 
to report changes in the residents' conditions. 

PART 2 General Comments 

1. Do you generally kn^w who is in the ESL program ? 

OxYes 0 No 

2. How would you rate his/her ability to use Enalish on the job ? 

O Excellent 0 Good 0 Average 0 Somewhat weak 0 Weak 

3. Have you noticed any additional changes since he/she enrolled 
in the ESL program ? (e.g. more confident) & ( "H Lc l> 



4. According to you, how much has the ESL program contributed to 
improving his/her languaqe skills ? _ 

O No idea O A lot 0 ^^ittle^g Not at all 

5. How can the ESL program better help nursing assistants improve ^ 
their language skills ? FrPz&T gjg ^/Uydw^. ^^Z ^-^^-SZ 

6. VJhat else might be done by supervisors, and by nursina 
assistants to help nursing assistants use English to perform 
their jobs ? J y {' I luXa^ f-v*-^- [aJ n^-^ 

THANK YO'J FOR FILLING OUT THIS QUESTIONNAIRE. IF YOU HAVE OUESTIOME , 
PLEASE CONTACT BARBARA IRVING , HSIAO CHANG , OR JOHAN UVIN. 
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FLOW SHEET EXERCISE 



SCORING SHEET 



1. Month 

2. Year 

3. Resident name 

4. Resident room # 

5. Initials 1 

6. Initials 2 

7. Signature 

8. Shift (=D) 

9. Day (=Tuesday) 

10. Bed Bath (=N) 

11. Partial (=N) 

12. Whirlpool/Shower (= A) 

13. Shave (=T) 

14. Mouth/Denture Care (=T) 

15. Mouth Special Care (=N) 

16. Nails (=T) 

17. Grooming (=1) 

18. Dressed Day/Night (=A) 

19- Socks and shoes only (== N or T) 

20. Not dressed (=N) 

21. Ambulate (=2/A) 

22. Distance ( = 20 1 or 2o ft or 20) 

23. Device ( = V or 2/A) 

24. Geri/Wheel Chair ( = V or 1/A) 

25. Transfer •( circle 1 or 1/T) 

26. Device (= V or 1/T) 

27 . Bed or W/C Position (= N or I ) 

28. ROM ( = I or V) 

29. Bladder Continent Freq. (= 6-8/A r 6-8/T, or 6-8) 

30. Oncontinent Freq. (= 0 or N) 

(29 can be 0 and 30 6-8 if total = 6-8) 

31. Tbileted (= 6-8/A r 6-8/T. or 6-8) 

32. Bowel Continent Freq. (= 2/A r 2/T. or 2) 

33. Incontinent freq. (= 0 or N) 

(32 can be 0 and 33 2 if total = 2) 

34. Toileted freq.( = 6-8/A, 6-8/T. or 6-8) 
.35. Bladder Retraining (=N) 

36. Bowel Retraining (=N) 

37. Catheter Care (=N) 

38. Wandering (=N) 

39. Hoarding (=N) 

40. Noisy («N) 

41. Verbal abuse (=N) 

42. Physical abuse (=N) 

43. Uncooperative (= N or V) 

44. Other (=N) 

45. Preventive Skin Care (=N) 

46. Elbow/Heel Protector (=N) 

47. Brace/Splint (=N) 

48. Sheep Skin (=N) 

49. Other (abdominal pain) 

50. Other column * 
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TOTAL (=T) 



SCORE: T X 2 = 



Procedures 



1. Explain purpose of the study and this particular assessment 

2. Explain the procedures: 

2.1. Give narrative in Chinese orally (recorded) 
or in writing. 

2.2. Tell participants to enter the information 
from the narrative onto the flow sheet. 

2.3. Check comprehension. 

2.4. Ask participants to complete the task. 

2.5. Say there is no time limit. 

2.6. Give participants the opportunity to 
ask questions. 

2.7. Explain how the assessment will be analyzed 
and how the findings will be used stressing 
that the information is kept confidential. 

3. Start the assessment. 

4. Discuss the results with participants. 

5. Provide participants with the opportunity to meet 
with their teacher if further discussion is desired. 
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